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BESRL  Technical  Research  Reports  and  Technical  Research  Notes  are  intended 
for  sponsors  of  R&D  tasks  and  other  research  and  military  agencies.  Any  findings 
ready  for  implementation  at  the  time  of  publication  are  presented  in  the  latter  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recommenda¬ 
tions  for  official  action  normally  are  conveyed  to  appropriate  military  agencies  by 
briefing  or  Disposition  Form. 


FOREWORD 


The  SURVEILLANCE  SYSTEMS  Project  hat  at  its  objective  the  development  of  scientific 
research  data  bearing  on  the  extraction  of  information  from  imagery  and  the  products  of  other 
sensors,  and  the  efficient  storage,  retrieval,  and  transmission  of  information  with  an  advanced 
computerized  image  interpretation  facility.  Research  results  ar*  used  in  future  systems  design 
and  in  the  development  of  enhanced  techniques  and  procedures  for  all  phases  of  the  image 
interpretation  process  within  the  data  reduction  facility. 

The  MAN-COMPUTER  FUNCTIONS  Task  is  one  of  four  research  Tasks  established  in  the 
Support  Systems  Research  Division  of  BESRL  to  concentrate  on  operational  segments  of  the 
surveillance  system.  One  major  effort  of  the  Task  is  devoted  to  the  development  and  evaluation 
of  input/output  procedures  for  the  description  and  transmission  of  tactical  intelligence  information. 
The  present  study  concentrated  on  two  work  methods  of  rapid  screening  to  determine  the  presence 
or  absence  of  military  targets  and  work  methods  of  setting  priorities  for  detailed  interpretation  of 
selected  frames  of  imagery. 


J.  E.  UHLANER,  Director 
U.  $.  Army  Behavioral  Science 
Research  Laboratory 


RAPID  SCREENING  OF  TACTICAL  IMAGERY  AS  A  FUNCTION  OF  DISPLAY  TIME 


BRIEF 


Requirement: 

To  assess  the  effectiveness  in  the  rapid  screening  of  tactical  imagery  of  two  techniques  for 
selecting  a  limited  number  of  frames  of  high  intelligence  potential  for  detailed  interpretation  and 
to  determine  the  effects  of  time  allowed  per  frame  on  the  effectiveness  of  the  two  techniques. 


Procedure: 

Two  samples  of  image  interpreters  (N's  of  33  and  30),  each  sample  consisting  of  three 
matched  groups,  screened  three  sets  of  imagery  at  three  display  time  intervals-5,  15,  and  25 
seonds  per  frame  for  Sample  1,  and  10,  20,  and  30  seconds  per  frame  for  Sample  2.  Subjects 
scanned  each  print  for  the  prescribed  time  while  performing  two  screening  functions:  (1)  annotat¬ 
ing  on  the  frame  all  areas  of  military  activity  detected,  and  (2)  assigning  priority  ratings  of  High, 
Medium,  and  Low  to  indicate  the  estimated  intelligence  value  of  the  frame.  The  two  methods  were 
compared  in  terms  of  accuracy  rate  and  validity  at  the  different  display  intervals. . 


Findings: 

Priority  ratings  were  better  indexes  of  the  information  potential  of  the  frames  than  were  the 
annotations.  The  ratings,  of  high  accuracy  even  with  short  viewing  fime,  improved  both  in  accu¬ 
racy  and  in  validity  with  longer  display  time.  The  validity  of  the  ratings  was  higher  with  sets  of 
imagery  characterized  by  relatively  few  target  areos  and  less  complex  background. 

The  validity  of  the  number  of  annotations  on  a  frame,  generally  low,  did  not  increase  with 
increased  display  time.  More  incorrect  as  well  as  correct  annotations  were  made. 


Utilization  of  Findings: 

The  priority  rating  method  of  screening  is  sufficiently  accurate  to  be  useful  even  where  only 
very  short  screening  time  can  be  allowed.  However,  imagery  differing  in  complexity  may  require 
adjustments  in  screening  time. 
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BACKGROUND  AND  OBJECTIVES 

Consistent  with  efforts  to  improve  the  reaction  time  of  military 
forces,  the  Army  is  developing  mobile  tactical  image  interpretation 
facilities  (TUT)  designed  to  help  speed  the  flow  of  intelligence  infor¬ 
mation  obtained  by  aerial  surveillance.  As  a  consequence  of  rapid  ad¬ 
vances  in  aerial  surveillance  system  technology,  including  the  develop¬ 
ment  of  telemetry  and  multi-sensor  platforms,  the  amount  of  imagery  to 
be  interpreted  by  a  TIIF  can  reach  overwhelming  proportions  at  peak  load 
times.  Future  TlIF's  may  often  be  required  to  process  large  amounts  of 
Imagery  with  great  rapidity,  at  the  same  time  maintaining  acceptable 
levels  Oi accuracy  and  completeness. 

The  projected  volume  of  imagery  raises  human  factors  problems  for 
both  the  TIIF  team  chief  and  individual  interpreters :  What  is  the  optimal 
method  of  processing  the  imagery?  Should  it  be  screened  first,  then 
Interpreted?  If  the  imagery  is  screened,  what  is  the  beat  method  of 
identifying  frames  of  greatest  potential  for  later  interpretation?  What 
realistic  workloads  can  be  assigned?  What  are  realistic  screening  rates? 
Are  such  rates  differential,  depending  on  the  characteristics  of  the 
Imagery?  The  general  purpose  of  the  present  study  was  to  investigate 
one  of  these  problems,  namely,  the  effect  of  variations  in  display  time— 
that  is,  rate  of  screening— on  interpreter  performance  in  screening 
tactical  imagery. 

Screening  is  the  process  of  selecting  from  a  mass  of  Imagery  those 
frames  which  have  high  intelligence  value  for  subsequent  more  detailed 
interpretation.  As  defined  here,  screening  does  not  include  or  exclude 
the  process  of  mentally  identifying  specific  targets  -  -suspicion  that  a 
frame  contains  targets  or  at  least  signs  of  military  activity  would  pre¬ 
sumably  underlie  its  selection  for  further  study.  The  desired  result  of 
the  screening  process,  however,  is  not  specific  identifications  but  a 
set  of  photographs  with  greater  than  average  information  potential  or 
intelligence  value. 

The  procedure  for  screening  tactical  Imagery  used  in  the  study  was 
divided  into  two  activities.  First,  the  scree ner  annotated  areas  of 
suspected  military  activity  directly  on  the  frame  of  imagery  be  was 
examining.  He  then  assigned  a  priority  rating  based  on  the  estimated 
intelligence  value  of  the  frame.  Both  actions  were  accomplished  by  the 
scree  ner  within  fixed  time  periods  ranging  from  5  to  30  seconds  per  frame. 
The  principal  objectives  of  the  experimentation  were  (1)  to  assess  the 
accuracy  and  validity  of  the  annotations  and  of  the  priority  ratings,  and 
(2)  to  determine  the  effect  of  variations  in  display  time  on  screening 
performance . 


METHOD 


Subject* 

Two  samples  of  Image  Interpreters  about  to  graduate  from  the  Image 
Interpretation  Course  at  the  U.  S.  Army  Intelligence  School  at  Fort 
Holabird,  Maryland  served  sis  subjects.  Each  sample  was  divided  Into 
three  groups  matched  on  rights  and  wrongs  scores  on  three  standard  per¬ 
formance  measures.  Sample  1  contained  11  subjects  In  each  group. 
Sample  2  contained  10  subjects  In  each  group,  making  a  total  N  of  63. 


Performance  Measure* 

The  three  performance  measures  used  In  the  study  were  selected  to 
provide  contrasting  geographical  and  terrain  conditions.  The  set  of 
Imagery  for  each  measure  consisted  of  15  conventional  black  and  white 
photographs  typical  of  the  "hard  copy”  with  which  the  Image  interpreter 
la  familiar.  All  photographs  In  a  given  set  were  taken  from  the  same 
flight  line.  Scale  on  the  three  sets  of  photographs  was  reasonably 
similar. 

Performance  Measure  T-lT^t  Imagery  cove,  ed  the  coast  of  Florida. 
Terrain  conditions  were  fairly  homogeneous,  consisting  of  heavily  wooded 
areas  with  small  clearings .  Terrain  was  intersected  by  Improved  and 
unimproved  roads.  Scale  warn  1  :  8,400. 

Performance  Measure  T-l8.  Imagery,  taken  during  World  War  II, 
covered  the  Bastogne  area  along  the  Luxembourg -Germany  border.  Photos 
were  taken  after  a  heavy  snowfall.  Terrain  features  included  groups  of 
buildings,  lightly  wooded  areas,  with  some  roads.  Scale  was  1  :  9*300* 

Performance  Measure  T-21.  Photos  of  the  Vire-Mortain,  Brittany, 
were  also  taken  during  World  War  II.  Terrain  was  typically  agricultural, 
with  numerous  wthaII  plots  of  land  surrounded  by  trees  or  hedgerows  and 
presenting  a  mosaic-like  appearance.  Several  photos  showed  heavily 
wooded  areas.  Scale  was  1  :  10,000. 

Apparent  military  activity  in  the  Imagery  was  denoted  by  vehicle 
trackage,  wheeled  and  tracked  vehicles,  gun  positions,  observation  and 
control  points,  (hjects  and  features  varied  considerably  within  a  given 
set  and  between  sets.  The  number  of  areas  of  military  activity  In  each 
print  In  each  set  is  presented  in  Appendix  A. 


^Designations  show  the  location  of  the  imagery  In  the  BESRL  library  of 
image  Interpretation  performance  measures. 
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Research  Design 


The  three  performance  measures  were  administered  to  each  sample. 

The  three  groups  within  a  sample  were  all  administered  the  same  three 
sets  of  imagery  but  In  a  different  order  and  with  different  display 
intervals— 5;  15#  and  25  seconds  per  frame  for  Sample  1#  and  10,  20, 
and  50  seconds  per  frame  for  Sample  2.  All  other  aspects  of  the  experi¬ 
ment  were  the  same  for  the  two  samples.  Also  of  seme  concern  was  the 
possible  practice  effect  which  might  Influence  performance  In  the  three 
trials.  The  design  employed,  therefore,  was  a  3  x  3  x  J  x  j  Graeco- 
Latin  square— display  time  x  performance  measures  x  order  (group)  x 
trials.  The  overall  designs  far  the  two  samples  are  outlined  In  Figure  1. 


Sample  1 


Order 

Trials 

1 

2 

3 

Group 

A 

T-17 

5  secs 

T-lfl 
15  secs 

T-21 

25  secs 

Group 

B 

T-l8 
25  secs 

T-21 

5  secs 

T-17 

15  secs 

Group 

C 

T-21 
15  secs 

T-17 
25  secs 

T-lfl 

5  secs 

Figure  1.  Graeco-Latin  square  experimental 


Sample  2 


Order 

Trials 

1 

2 

3 

Group 

A 

T-17 
10  secs 

T-lfl 
20  secs 

T-21 

30  secs 

Group 

B 

T-lB 
30  secs 

T-21 
10  secs 

T-17 

20  secs 

Group 

C 

T-21 
20  B6C8 

T-17 
30  secs 

T-lfl 

10  secs 

design  for  Samples  1  and  2 


Screening  Procedures 

Each  group  was  instructed  to  scan  each  print  In  a  set  for  the  pre¬ 
scribed  display  time.  During  this  time,  each  Interpreter  first  annotated 
all  areas  of  military  activity  that  he  detected  on  the  print  by  drawing 
circles  around  each  area,  restricting  the  size  of  the  circle  to  the  area 
of  activity  as  much  as  possible.  He  then  assigned  to  each  print  a 
priority  rating  of  High,  Medium,  or  Low,  based  an  the  potential  intelli¬ 
gence  Information  contained  in  the  print.  Before  administration  of  a 
set  of  Imagery,  the  experimenter  demonstrated  the  duration  of  the  time 
Interval  each  frame  would  be  displayed.  All  administrations  were  timed 
by  stop  watch.  The  complete  Instructions  cure  given  in  Appendix  B. 
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D<*p«nd«n  Variable* 

The  main  concern  of  the  experiment  was  the  accuracy  and  validity  of 
the  annotations  and  priority  ratings  across  the  six  screening  time  inter¬ 
vals.  Differences  attributable  to  the  groups  within  a  sample  were  of 
little  concern.  However,  the  influence  of  the  performance  measures  (the 
sets  of  imagery)  and  of  the  trials  (the  practice  effect)  on  screening 
performance  was  of  Interest.  The  effect  of  the  four  experimental 
conditions  — time ,  group,  set  of  Imagery,  trial— was  determined  for  seven 
dependent  variables: 

1.  Total  number  of  correct  annotations,  sunned  across  15  frames  in 

a  set. 

2.  Completeness  of  annotations.  Total  number  of  correct  annotations 
divided  by  the  total  number  of  areas  of  military  activity  in  15  frames. 

3.  Total  number  of  incorrect  annotations. 

4.  Accuracy  of  annotations.  Total  number  of  correct  annotations 
divided  by  the  total  number  of  annotations  made  in  15  frames. 

5.  Validity  of  annotations.  Correlation,  across  15  frames,  between 
the  number  of  annotations  (correct  and  incorrect)  made  on  each  frame  by 
the  subject  and  the  number  of  areas  of  military  activity  on  the  frame. 

6.  Accuracy  of  priority  ratings.  Number  of  frames  for  which  the 
interpreter's  ratings  of  High,  Medium,  and  Low  were  the  same  as  that  of 
expert  interpreters  divided  by  15. 

7.  Validity  of  priority  ratings.  Correlation,  across  15  frames, 
between  the  priority  ratings  given  by  the  subject  and  the  number  of 
areas  of  military  activity  on  the  frame. 


Statistical  Analysis 

Analyses  of  variance  consistent  with  the  Graeco-Latin  square  design 
were  computed  for  the  seven  dependent  variables.  Since  the  two  samples 
were  matched,  the  mean  squares  of  the  two  samples  were  tested  for  signifi¬ 
cant  differences,  where  the  data  permitted.  Where  appropriate,  in  order 
to  provide  a  continuum  in  display  time  from  3  to  30  seconds,  the  data  for 
the  two  samples  were  combined  for  presentation  in  tabular  form.  This 
approach  facilitated  exploration  of  optimal  display  intervals  for  a  given 
dependent  variable.  Where  analysis  of  variance  of  correlation  coeffi¬ 
cients  was  computed,  the  coefficients  were  converted  using  the  r  to  z 
transformat  ion . 


t 
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RESULTS--ANN0TATI0NS  PROCEDURE 
Number  of  Correct  Annotation* 

The  mean  number  of  correct  annotations  generally  increased  as  the 
display  interval  increased  (Table  1).  This  result  was  consistent  with 
expectations,  since  the  longer  display  time  afforded  the  interpreter 
more  opportunity  to  look  at  suspected  areas  and  to  make  more  correct — as 
well  as  incorrect— annotations.  The  analysis  of  variance  for  both 
samples  yielded  highly  significant  results  for  screening  time.  Indicating 
that  significantly  different  numbers  of  correct  annotations  were  made  for 
the  different  time  intervals  (See  Appendix  C).  There  were  no  significant 
differences  in  correct  annotations  attributable  to  the  other  main  effects 
(group,  trial,  sets  of  imagery).  In  none  of  the  analyses  conducted  for 
the  other  dependent  variables  were  significant  differences  obtained  which 
could  be  attributed  to  group  or  trial  effects.  There  were,  however,  sig¬ 
nificant  performance  measure  effects  on  other  variables.  All  analysis  of 
variance  data  are  presented  in  Appendix  C. 


Table  1 

MEAN  NUMBER  CF  CORRECT  ANNOTATIONS  AT  EACH  DISPLAY  INTERVAL 

FOR  SAMPLES  1  AND  2 


Performance 

Measure 

Display  Time* 

5  sec. 

10  sec. 

13  sec. 

20  sec. 

25  sec. 

30  sec. 

T-17 

9.6 

12.8 

13.2 

13-7 

20.4 

19.4 

T-lB 

9-3 

9-2 

14.5 

17.1 

17.5 

14.7 

T-21 

12.2 

1.9 

15.5 

11.6 

16.4 

18.9 

Mean 

10.3 

8.0 

14.4 

14.1 

18.1 

17.7 

•P  <  .01  (or  S,  15,  25  .Dcondi  (Sampl.  1)  and  for  10,  20,  10 


nda  (Sampla  2). 


Completed***  of  Annotation* 


Of  additional  interest  is  the  question,  "Of  the  total  amount  of 
military  activity  present  in  the  imagery,  what  percentages  were  annotated 
at  each  of  the  various  display  intervals?"  In  the  results,  completeness 
varied  significantly  as  a  function  of  screening  time  and  performance 
measure.  Table  2  shows  that  mean  completeness  across  the  three  measures 
tended  to  increase  with  an  increase  in  display  time. 


-  5  - 


A  highly  significant  main  effect  for  performance  measures  reflected 
the  difference  in  tbs  percentages  of  total  information  extracted  from 
tbs  three  different  sets  of  imagery  constituting  the  performance  measures. 
It  ted  not  been  possible  to  match  the  imagery  in  the  different  sets  in 
terms  of  the  nusfcer  of  areas  of  military  activity  shown.  Tab lp  3  shows 
that  the  more  target  areas  in  a  given  set,  the  lower  the  mean  complete¬ 
ness.  On  the  surface,  the  significant  differences  in  completeness  would 
seem  attributable  either  to  differences  in  the  number  of  areas  of  activity 
in  the  imagery  set  or  to  the  complexity  of  the  Imagery— or  to  the  inter¬ 
action  of  the  two  factors. 


Table  2 

MEAN  COMPLETENESS  CF  ANNOTATIONS  FCR  SAMPLES  1  AND  2 


Performance 

Measure** 

Display  Time* 

5  sec. 

10  sec. 

15  sec. 

20  sec. 

25  sec. 

30  sec. 

T-17 

.14 

.18 

.19 

.20 

.29 

.28 

T-l8 

.09 

.09 

.14 

.l£ 

.17 

.14 

T-21 

.09 

.01 

.11 

.08 

.12 

.14 

Mean 

.11 

.10 

.1^ 

.15 

.19 

.19 

•P  <  .01  for  5,  15,  25  seconds  (Sample  1)  and  for  10, 
••P  <  .01  for  performance  In  both  samples. 

20,  30  seconds  (Sample  2). 

Table  3 

MEAN  COMPLETENESS  QF  ANNOTATIONS  FCR  SETS  CF  IMAGERY  WITH 

VARYING  NUMBERS  QF  TARGET  AREAS 

Performance 

Mean 

No.  of  Areas  of 

Mean  No.  of  Areas 

Measure 

Completeness 

Military  Activity 

Correctly  Ident. 

T-17 

.21 

69 

14.9 

T-l8 

.13 

104 

13.9 

T-21 

.09 

134 

12.7 
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Number  of  Incorrect  Annotations 


The  analyses  of  variance  for  both  samples  yielded  significant 
differences  In  number  of  Incorrect  annotations  for  varying  display 
intervals.  The  mean  number  of  Incorrect  annotations  generally  in¬ 
creased  with  Increased  time  (Table  If).  This  finding  Is  similar  to 
previous  BESRL  findings  for  unspeeded  interpretation  performance . 

The  longer  Interpreters  look  at  Imagery,  the  greater  the  total  number 
of  responses — correct  and  incorrect — they  make. 


Table  4 

MEAN  NUMBER  CF  INCORRECT  ANNOTATIONS  FOR  SAMPLES  1  AND  2 


Performance 

Measure 

Display  Time* 

5  sec. 

10  sec. 

15  sec. 

20  sec. 

25  sec. 

30  sec. 

T-17 

9.09 

12.40 

20.27 

20.80 

26.45 

27.80 

T-l8 

16.36 

3.20 

17.55 

17.40 

29.91 

19.60 

T-21 

lB.08 

20.00 

16.91 

15.40 

32.00 

30.00 

Mean 

14.55 

11.80 

18.24 

17.99 

29.45 

25.80 

•P  <  .01  for  5.  15,  25  seconds  (Sample  1)  nnd  for  10,  20,  30  seconds  (Sample  2). 


Accuracy  of  Annotations 


In  the  analysis  of  variance  for  this  variable,  neither  sample  pro¬ 
duced  any  significant  main  effects.  Such  a  finding  Indicates  ttat  mean 
accuracy  of  annotation  In  screening  Is  not  affected  by  variations  in 
display  time.  Essentially,  the  annotations  made  during  a  5-  or  10-second 
screening  Interval  were  Just  as  accurate  as  those  made  during  a  25-  or 
30-second  interval  (see  Table  5).  Similarly,  there  was  little  difference 
in  the  mean  accuracy  for  the  different  performance  measures. 
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Table  5 

MEAN  ACCURACY  CF  ANNOTATIONS  FOR  SAMPLES  1  AND  2 


Performance 

Measure 

Display  Time 

3  sec. 

10  sec. 

13  sec. 

20  sec. 

23  sec. 

30  sec. 

T-17 

.51 

.51 

.37 

.42 

.46 

.43 

T-18 

.37 

.32 

.47 

•51 

.39 

.46 

T-21 

.46 

.35 

•  50 

.47 

.49 

•  45 

Mean 

.44 

.39 

.44 

.47 

.44 

.44 

Validity  of  Total  Number  of  Annotations 

The  validity  of  the  annotations  was  measured  for  each  interpreter 
by  the  correlation  between  the  total  number  of  annotations  made  on  each 
of  the  15  frames  in  a  performance  measure  and  the  number  of  areas  of 
military  activity  actually  on  the  frame.  In  general,  the  obtained  coef¬ 
ficients  were  low,  especially  those  obtained  in  Sample  2  at  10,  20,  and 
30  seconds  display  time  (Table  6).  There  was  no  evidence  of  significant 
variation  in  validity  as  a  function  of  display  time  in  either  sample. 

The  validity  index  reflects  the  extent  to  which  the  number  of  annotations 
recorded  is  proportional  to  the  number  of  actual  areas.  Thus,  the  slow 
or  cautious  subject  was  not  necessarily  penalized  as  compared  to  the 
faster  subject  whose  average  number  of  annotations  may  have  more  closely 
approached  the  actual  number  of  areas  on  the  frames. 


Table  6 


MEAN  VALIDITY  COEFFICIENTS  CF  TOTAL  NUMBER  CF  ANNOTATIONS 

FCR  SAMPLES  1  AND  2 


Performance 

Measure* 

Display  Time 

5  sec. 

10  sec. 

15  sec. 

20  sec. 

23  sec. 

30  sec. 

T-17 

.45 

.06 

.43 

-.24 

•  53 

-.22 

T-18 

.21 

.04 

•  39 

-.01 

•  36 

-.06 

T-21 

.13 

.08 

.02 

-.12 

-.06 

K"\ 

O 

e 

1 

Mean 

.26 

.06 

.28 

-.12 

CO 

CM 

• 

-.10 

•P  ^  .01  for  performance  measures  In  both  samples. 
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Validity  coefficients  obtained  for  performance  measure  T-21  were 
significantly  lower  than  those  obtained  for  the  other  two  measures— 
practically  zero,  in  fact,  for  all  display  intervals .  Although  T-21 
hart  the  largest  number  of  areas  of  military  activity,  the  mean  perform¬ 
ance  of  the  subjects  in  number  of  correct  and  incorrect  annotations  was 
poorest  on  this  measure.  As  indicated  earlier,  the  Images  comprising 
T-21  were  more  complex  or  heterogeneous  in  content  than  were  Images  in 
the  other  two  measures. 


RESULTS--PRIORITY  RATINGS  PROCEDURE 

Under  this  method  of  screening,  it  was  assumed  that  in  the  quick- 
time  or  near  real-time  situation,  the  image  interpreter  is  presented 
with  a  large  number  of  frames  of  imagery,  each  frame  being  displayed 
for  a  brief  time.  The  interpreter  has  to  decide  very  quickly  whether  a 
frame  contains  sufficient  information  to  be  earmarked  for  further  analy¬ 
sis  and  interpretation.  The  primary  question  which  arises  in  this  regard 
is,  "Whit  effect  does  the  length  of  time  an  Image  is  displayed  have  on 
the  accuracy  and  validity  with  which  frames  are  assigned  priorities  V" 


Accuracy  of  Priority  Ratings 


The  accuracy  with  which  the  interpreters  assigned  ratings  of  High, 
Medium,  and  Low  to  prints  of  tactical  Imagery  did  not  vary  significantly 
for  display  intervals  of  5,  15,  and  25  seconds  (Sample  l);  however,  there 
were  significant  differences  in  the  rating  accuracy  for  display  times  of 
10,  20,  and  50  seconds — and  for  the  different  performance  measures  as 
well  (Table  7).  Note  that  the  ratings  tended  to  be  more  accurate  for  the 
longer  display  times — 20,  25,  and  50  seconds.  Also,  the  ratings  assigned 
to  performance  measure  T-17,  the  set  of  Images  containing  the  fewest  areas 
of  military  activity,  tended  to  be  more  accurate  than  ratings  assigned  to 
the  other  sets. 


Table  7 


MEAN  ACCURACY  CF  PRIORITY  RATINGS  FCR  SAMPIZS  1  AND  2 


Performance 

Measure** 

Display  Time* 

5  sec. 

10  sec. 

15  sec. 

20  sec. 

25  sec. 

50  sec. 

T-17 

.53 

•  59 

.49 

.62 

.65 

•59 

T-lB 

.42 

.48 

.49 

.45 

.50 

.58 

T-21 

.50 

.36 

.45 

•51 

.53 

.58 

Mean 

.48 

.47 

.48 

•  53 

•  55 

.58 

•P  <.  .01  for  10,  20,  and  30  seconds  (Sample  2). 
•*P  ^  .01  for  performance  measures  for  Sample  2. 
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Validity  of  Priority  Ratings 

The  validity  of  the  priority  ratings  was  measured  by  the  correla¬ 
tion  between  the  ratings  made  by  the  subjects  and  the  number  of  areas  of 
military  activity  actually  on  the  frames.  Unlike  the  validity  of  the 
annotations,  the  validity  of  the  ratings  tended  to  increase  with  in¬ 
creased  display  time  (Table  8).  Also  unlike  the  annotations,  the  va¬ 
lidity  coefficients  for  ratings  in  Sample  2  were  not  markedly  different 
from  those  in  Sample  1.  However,  as  with  the  annotations,  the  validity 
coefficients  of  the  priority  ratings  for  Performance  Measure  T-21  were 
close  to  zero. 


Table  8 

MEAN  VALIDITY  COEFFICIENTS  QF  PRIORITY  RATINGS 
FCR  SAMPLES  1  AND  2 


Display  Time* 

Performance  - 


Measure** 

5  sec. 

10  sec. 

15  sec. 

20  sec. 

25  sec. 

30  sec. 

T-17 

.38 

.1*4 

.31 

•  58 

• 

ON 

O 

.56 

T-18 

.44 

.33 

.30 

.35 

.45 

.52 

T-21 

-.11 

-.19 

-.l£ 

KN 

O 

• 

1 

-.08 

.00 

Mean 

.24 

.19 

.15 

.30 

.32 

.36 

•P  <  .05  for  5,  15, 

25  aeconda  (Sample  1)  and  for  10 

,  20,  30  aaconds  (Sample  2). 

••P  4C  .01  for  performance  measures  In  both  aamplea. 


SUMMARY  OF  RESULTS 

1.  Both  the  number  of  correct  and  the  number  of  Incorrect  annota¬ 
tions  made  by  the  Interpreters  significantly  Increased  with  an  Increase 
in  display  time. 

2.  The  completeness  of  the  annotations  made  by  the  interpreters 
increased  significantly  with  an  increase  in  screening  time,  and  also 
varied  significantly  for  the  different  performance  measures. 

3»  The  accuracy  of  annotations  and  the  validity  of  the  total  number 
of  annotations  made  by  the  interpreters  were  not  influenced  by  variations 
in  screening  time. 


i 
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4.  The  accuracy  of  the  priority  ratings  significantly  increased 
over  screening  times  of  10,  20,  and  20  seconds,  but  not  over  screening 
times  of  5,  10,  and  12  seconds. 

5.  The  validity  of  the  priority  ratings  increased  with  an  increase 
in  screening  time,  and  also  varied  significantly  for  the  different  per¬ 
formance  measures. 


IMPLICATIONS  OF  THE  FINDINGS 

Results  definitely  favored  the  priority  ratings  over  the  annotation 
screening  technique.  In  general,  the  priority  ratings  provided  more 
valid  Indexes  of  the  number  of  areas  of  military  activity  in  the  frames 
than  did  the  annotations .  Although  the  number  of  correct  annotations 
increased  with  display  time,  the  fact  that  the  number  of  incorrect 
annotations  Increased  as  well  allowed  no  overall  gain  in  accuracy.  The 
validity  of  the  total  number  of  annotations  was  generally  low,  and  did 
not  vary  significantly  with  screening  time.  This  result  is  particularly 
important  since  total  number  of  annotations  made  on  a  frame  might  have 
been  considered  a  valid  index  of  the  value  of  the  frame.  The  priority 
ratings,  on  the  other  hand,  improved  both  in  accuracy  and  validity  with 
longer  display  time.  The  ratings  were  surprisingly  accurate  in  both 
samples,  even  for  the  short  display  times. 

Note  that  the  scale  of  the  Imagery  used  in  the  study  was  somewhat 
smaller  than  is  much  current  operational  tactical  imagery.  In  view  of 
the  high  validity  of  ratings  obtained  with  the  imagery  used,  investiga¬ 
tion  of  ratings  could  well  be  extended  to  Imagery  of  larger  scale. 
Screening  time  of  less  than  2  seconds  may  be  effective  with  large-scale 
frames .  Also,  the  use  of  rating  scales  with  more  than  the  three  points 
of  High,  Medium,  and  Low  would  be  expected  to  increase  the  reliability 
and  hence  the  validity  of  the  priority  assignments. 

With  regard  to  the  consistent  differences  in  screening  performance 
found  among  the  three  performance  measures,  the  inference  is  that  the 
differences  were  due  for  the  most  part  either  to  the  complexity  of  the 
background  in  the  imagery  or  to  the  varying  numbers  of  military  activity 
areas  (since  quality  and  scale  were  similar).  Generally,  performance 
was  better  on  the  two  measures  which  were  less  complex  and  showed  fewer 
target  areas  (T-17  and  T-l8)  than  on  the  third  measure  (T-21).  The 
prints  in  T-21  presented  a  mosaic-like  appearance  due  to  the  numerous 
plots  of  farmland.  The  numerous  edges  presented  to  the  eye  constituted 
a  natural  partitioning  of  the  display  and  ®ny  have  distracted  the  inter¬ 
preters.  Longer  display  times  than  were  used  in  the  study  are  probably 
necessary  for  scanning  complex  imagery. 


Major  Implications  with  regard  to  the  ratings  are  as  follows : 

1.  Although  the  accuracy  and  validity  of  ratings  varied  signifi¬ 
cantly  among  different  sets  of  Imagery,  the  absolute  levels  of  these 
indexes  were  sufficiently  promising,  even  for  very  short  screening 
times,  to  suggest  the  use  of  such  ratings  in  some  operational  screening 
tasks. 

2.  Considering  screening  times  of  50  seconds  or  less,  the  longer 
the  time  allowed  for  screening,  the  more  accurate  and  valid  are  inter¬ 
preters'  ratings  of  the  potential  intelligence  value  of  frames  of 
Imagery. 

5-  More  refined  rating  procedures,  a  wider  sample  of  imagery,  and 
a  greater  range  of  display  and  viewing  conditions  might  well  yield  more 
definitive  results  with  regard  to  optimum  display  time. 
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APPENDIX  A 


DISTRIBUTION  OF  NUMBERS  CF  AREAS  CF  MILITARY  ACTIVITY 
BY  SETS  CF  IMAGERY 


Print  Number 

2liZ 

T-18 

T-21 

1 

0 

5 

11 

2 

0 

16 

18 

3 

0 

2 

9 

4 

0 

3 

5 

5 

10 

2 

14 

6 

14 

18 

0 

7 

10 

10 

16 

8 

3 

8 

11 

9 

2 

8 

4 

10 

1 

10 

6 

11 

9 

7 

2 

22 

7 

4 

6 

13 

5 

5 

3 

16 

3 

3 

15 

15 

5 

3 

14 

L 

69 

104 

134 
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APPENDIX  B 


GENERAL  INSTRUCTIONS  TO  EXAMINEES 


Distribute  set  of  imagery  T-  -S.  Then  say: 

PUT  YOUR  NAME  ON  THE  ENVELOP.  NOW  YOU  WILL  HAVE  5  SECONDS  TO 
ACCOMPLISH  THE  STEPS  1REVIOUSLY  DESCRIBED  FCR  EACH  PRINT  IN  THE  PACKET. 
BEFORE  YOU  BEGIN,  LET  ME  GIVE  YOU  AN  IDEA  CF  HCW  LONG  5  SECONDS  ARE. 

I'LL  START  THE  WATCH  NOW.  Pause  for  5  seconds.  Then  say: 

STOP!  BY  THIS  TIME  YOU  SHOULD  HAVE  MADE  YOUR  ANNOTATIONS  AND  * .  -R 
JUDGMENT  CF  HIGH,  MEDIUM,  CR  LOW  PRIORITY.  ALL  RIGHT,  PIACE  THE  FIRST 
PHOTO  IN  FRONT  CF  YOU.  READY?  BEGIN! 

After  5  seconds  have  elapsed,  say:  STOP!  TAKE  YOUR  NEXT  PRINT. 
READY?  BEGIN! 

After  5  seconds  have  elapsed,  say:  STOP!  Continue  this  procedure 
until  all  fifteen  prints  have  been  completed;  then  have  them  replace  the 
prints  in  the  packet. 


Distribute  the  next  packeo,  T-  -S.  Then  say: 

POT  YOUR  NAME  ON  THE  ENVELOP.  NOW  YOU  WILL  HAVE  15  SECONDS  TO 
ACCOMPLISH  THE  STEPS  PREVIOUSLY  DESCRIBED  FOR  EACH  PRINT  IN  THE  PACKET. 
BEFORE  YOU  BEGIN,  LET  ME  GIVE  YOU  AN  IDEA  CF  HOW  LONG  15  SECONDS  ARE. 
I'LL  START  THE  WATCH  NCW.  Pause  for  15  seconds.  Then  say: 

STOP!  BY  THIS  TIME  YOU  SHOULD  HAVE  MADE  YOUR  ANNOTATIONS  AND  YOUR 
JUDGMENT  CF  HIGH,  MEDIUM,  CR  LOW  PRIORITY.  ALL  RIGHT,  PLACE  THE  FIRST 
PHOTO  IN  FRONT  CF  YOU.  READY?  BEGIN! 

After  15  seconds  have  elapsed,  say:  STOP!  TAKE  UP  YOUR  NEXT  LRINT. 
READY?  BEGIN! 

After  15  seconds  have  elapsed,  say:  STOP!  Continue  this  procedure 
until  all  fifteen  prints  have  been  completed*,  then  have  them  replace  the 
prints  in  the  packet. 


Distribute  the  next  packet,  T-  -S.  Then  say: 

POT  YOUR  NAME  ON  THE  ENVELOP.  NOW  YOU  WILL  HAVE  25  SECONDS  TO 
ACCOMPLISH  THE  STEPS  PREVIOUSLY  DESCRIBED  FOR  EACH  PRINT  IN  THE  PACKET. 
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BEFORE  YOU  BEGIN,  LET  ME  GIVE  YOU  AN  IDEA  CF  HOW  LONG  25  SECONDS  ARE. 
I'LL  START  THE  WATCH  NOW.  Pause  for  25  seconds.  Then  say: 

STOPJ  BY  THIS  TIME  YOU  SHOUID  HAVE  MADE  YOUR  ANNOTATIONS  AND  YOUR 
JUDGMENT  CF  HIGH,  MEDIUM,  OR  LOW  PRIORITY.  ALL  RIGHT,  PLACE  THE  FIRST 
PHOTO  IN  FRONT  OF  YOU.  READY?  BEGIN! 

After  25  seconds  have  elapsed,  say:  STOP!  TAKE  YOUR  NEXT  PRINT. 
READY?  BEGIN! 

After  25  seconds  have  elapsed,  say:  STOP!  Continue  this  procedure 
until  all  fifteen  prints  have  been  completed^  then  have  them  replace  tbs 
prints  in  \be  packet. 


APPENDIX  C 


ANALYSIS  QF  VARIANCE  TABIES 


Table  C-l 

SUMMARY  CF  ANALYSIS  CF  VARIANCE— NUMBER  CF  CCRRECT  ANNOTATIONS 


Source 

Sample  1 

Sample  2 

df 

ms 

f 

df 

ms 

f 

Between  Subjects 

52 

64.5 

29 

97.9 

Groups 

2 

20.5 

.304 

2 

285.7 

5.599 

Subjects  W  Groups 

50 

67.4 

27 

84.0 

Within  Subjects 

66 

49.1 

60 

55.6 

Trials 

2 

61.4 

1.759 

2 

10.7 

.404 

Sets 

2 

7.9 

.215 

2 

155.7 

5.855* 

Screening  Times 

2 

492.9 

15.963* 

2 

723.0 

27.192* 

Error  Within 

6o 

55.5 

54 

26.6 

TOTAL 

96 

54.1 

89 

68.0 

*P  <  .01 
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Table  C-2 


SUMMARY  CF  ANALYSIS  CF  VARIANCE— CCMPIETE  NESS  CF  ANNOTATIONS 


Source 

Sample  1 

Sample  2 

df 

MS 

F 

df 

MS 

F 

Between  Subjects 

32 

.007 

29 

.009 

Groups 

2 

.009 

1.286 

2 

.016 

1.778 

Subjects  W  Groups 

30 

.007 

27 

.009 

Within  Subjects 

66 

.008 

60 

.010 

Trials 

2 

.015 

4.08 

2 

.002 

.667 

Sets 

2 

.089 

24.18* 

2 

.152 

50.667* 

Screening  Time 

2 

.064 

17.39* 

2 

.064 

21.333* 

Error  Within 

60 

.004 

54 

8 

• 

TOTAL 

98 

.007 

89 

.010 

*P  <  .001 


Table  C-3 


SUMMARY  CF  ANALYSIS  CF  VARIANCE— NUMBER  CF  INCORRECT  ANNOTATIONS 


Source 

Between  Subjects 
Groups 

Subjects  W  Groups 

Within  Subjects 
Trials 
Sets 

Screening  Tine 


Sample  1 


df  MS  F 


32 

374.8 

2 

123.3 

.31 

30 

391.6 

66 

167.5 

2 

11*8.5 

1.36 

2 

104.1 

•95 

2 

1989.1 

18.12* 

Sample  2 

df  MS  ] 


29 

2 

27 

603.9 

553.3 

X.09 

60 

2 

269.9 

3.72 

2 

258.8 

2.88 

2 

1465.4 

20.17* 

Error  Within 


6o  109.5 


54  72.6 


TOTAL 
*P  <  .01 


98 


235-2 


89 


269.2 
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Table  C-4 


SUMMARY  CF  ANALYSIS  CF  VARIANCE  —ACCURACY  CF  ANNOTATIONS 


Source 

Sample  1 

Sample  2 

df 

MS 

F 

uf 

MS 

F 

Between  Sublets 

32 

.041 

32 

.075 

Groups 

2 

e 

o 

vn 

.8631 

2 

.068 

.8679 

Subjects  W  Groups 

30 

.041 

30 

.078 

Within  Subjects 

66 

.035 

66 

CM 

O 

• 

Trials 

2 

.006 

.1849 

2 

.05* 

1.7420 

Sets 

2 

.049 

1.3850 

2 

.089 

.2751 

Screening  Tine 

2 

.028 

.8021 

2 

.041 

1.2767 

Error  Within 

6o 

.035 

60 

.032 

TOTAL 

98 

.064 

98 

.047 

Table  C-5 

SUMMARY  CF  ANALYSIS  CF  VARIANCE— ACCURACY  CF  PRIORITY  RATINGS 


Source 

Sample  1 

_ 1 

Sample  2 

df 

MS 

F 

df 

MS 

F 

Between  Subjects 

32 

.0004 

32 

.0005 

Groups 

2 

.0001 

.1931 

2 

.0001 

.2122 

Subjects  W  Groups 

30 

.0004 

30 

.0005 

Within  Subjects 

66 

.0006 

66 

.0006 

Trials 

2 

.0007 

1.3639 

2 

.0020 

4.9169 

Sets 

2 

.0012 

2.1699 

2 

.0029 

7.1582» 

Screening  Time 

2 

.0014 

2.6126 

2 

.0022 

5.4307* 

Error  Within 

60 

.0005 

60 

.0004 

TOTAL 

98 

.0005 

98 

.0006 

Table  C-6 


SUMMARY  Cf  ANALYSIS  <F  VARIANCE— VALIDITY  CF  ANNOTATIONS 


Source 

Sample  1 

Sample  2 

d f 

MS  F 

df 

MS 

F 

Between  Subjects 

Groups 

2 

.02  .23 

2 

.12 

1.66 

Subjects  W  Groups 

30 

.10 

27 

.07 

Within  Subjects 

Trials 

2 

.13  1.23 

2 

.01 

.12 

Sets 

2 

1.59  I5.5O** 

2 

.14 

I.76 

Screening  Time 

2 

.04  .40 

2 

.35 

4.34* 

Graeco-Ietin  Error 

0 

.25 

0 

.02 

Latins  x  S/G 

60 

.10 

54 

.08 

*P  <  .05 
**P  <  .01 


Table  C-7 


SUMMARY  CF  ANALYSIS  CF  VARIANCE— VALIDITY  CF  PRIORITY  RATINGS 


Source 

Sample  1 

Sample  2 

df 

MS 

F 

df 

MS 

F 

Between  Subjects 

Groups 

'  2 

.15 

1.43 

2 

.04 

.44 

Subjects  W  Groups 

30 

.10 

27 

.09 

Within  Subjects 

Trials 

2 

.04 

.60 

2 

.04 

.57 

Sets 

2 

3.62 

50.13** 

2 

3.62 

50.64** 

Screening  Time 

2 

•37 

5.08** 

2 

•31 

4.25* 

Graeco-Iatin  Error 

0 

.09 

0 

.08 

Latins  x  S/G 

60 

.07 

57 

.07 

*P  <  .05 
**P  <  .01 
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11  abstract  Rapid  advances  in  Army  aerial  surveillance  system  technology,  including 
the  development  of  telemetry  and  multi-sensor  platforms,  has  intensified  the  need 
for  development  of  improved  future  systems  design  as  well  as  enhanced  techniques  and| 
procedures  for  all  phases  of  the  image  interpretation  process.  The  principal  objec¬ 
tives  of  the  experimental  study  reported  in  Technical  Research  Dote  1&9  were  to  as¬ 
sess  the  effectiveness  of  two  techniques  for  rapid  screening  to  select  imagery  framesj 
of  high  military  potential  and  to  determine  the  effects  of  variations  in  display 
time  on  screening  performance.  Two  samples  of  image  interpreters,  each  consisting  of 
three  mat  died  groups,  screened  three  sets  of  imagery  at  three  different  display  time 
intervals  (5,  15,  and  25  seconds  per  frame  for  Sample  1;  10,  20,  and  50  seconds  per 
frame  for  Sample  2).  Each  interpreter  was  instructed  to  perform  two  screening  func¬ 
tions  while  scanning  each  print --1)  annotate  on  the  frame  all  areas  of  military  ac¬ 
tivity  he  detected;  and  2)  assign  to  each  print  a  priority  rating  of  High,  Medium, 
or  Low  to  indicate  estimated  intelligence  value  of  the  frame.  Interpreter  perform¬ 
ance  under  the  two  methods  was  compared  in  terms  of  accuracy  and  validity  of  the 
annotations  and  priority  ratings  across  the  six  screening  time  intervals.  Results  of 
the  study  definitely  favored  the  priority  ratings  technique  over  annotation  screen¬ 
ing.  The  ratings,  of  high  accuracy  even  with  short  viewing  time,  improved  signifi¬ 
cantly  both  in  accuracy  and  in  validity  with  longer  display  time.  Generally,  per¬ 
formance  was  better  on  the  measures  which  were  less  complex  and  showed  fewer  target 
areas.  Validity  of  the  number  of  annotations  on  a  frame,  generally  low,  did  not  vary| 
significantly  with  display  time.  More  incorrect  as  well  as  correct  annotations  were 
made,  a  finding  similar  to  previous  BESRL  findings  for  unspeeded  interpretation  per-| 
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